COGNITIVE  &  AFFECTIVE  PREDICTORS  OF  SIMULATION  PERFORMANCE 


Melissa  Dixon*,  Debra  Patton,  Linda  Fatkin,  Jock  Grynovicki,  Charles  Hernandez1 
U.S.  Army  Research  Laboratory,  Human  Research  &  Engineering  Directorate 
Aberdeen  Proving  Ground,  MD  21005-5425 
fFort  Sill,  OK,  73503-5600 


ABSTRACT 

The  following  paper  discusses  a  study  that 
investigated  how  select  cognitive  and  affective 
assessment  surveys  may  be  predictive  of  decision-making 
performance  in  a  training  simulation.  In  this  study.  Army 
Officers  completed  a  simulation  and  answered  cognitive 
assessment  surveys.  Results  showed  that  scores  on  the 
Uncertainty  Response  Scale,  Desire  for  Change  subscale 
(Greco  &  Roger,  2001),  were  predictive  of  overall 
performance.  As  predicted,  there  were  also  significant 
changes  in  affect,  as  measured  by  the  Multiple  Affect 
Adjective  Check  List  -  Revised  (Lubin  &  Zuckerman, 
1999),  on  all  subscales  in  response  to  key  events  within 
the  simulation.  These  affective  responses  are  indicative 
of  the  participants’  immersion  in  the  simulation  as  they 
assumed  the  role  of  the  main  character.  We  propose  the 
Cognitive  Assessment  Support  Tool  (CAST)  as  a 
methodology  for  incorporating  the  results  from  cognitive 
assessment  surveys  into  tailored  feedback  provided  to 
users  by  intelligent  tutors,  or  virtual  coaches,  during 
simulations.  The  results  and  the  proposed  CAST 
framework  are  discussed  in  the  context  of  the  guided 
experiential  learning  (GEL)  model  of  instruction 
(Kirschner,  Sweller,  &  Clark,  2006),  which  emphasizes 
providing  background  information  followed  by  practicing 
the  task  actions  and  instructional  feedback. 


1.  INTRODUCTION 

The  Training  and  Doctrine  Command  (TRADOC) 
Pamphlet  525-66  addresses  the  Military  Operations  Force 
Operating  Capabilities  (FOCs)  for  the  Future  Force 
Soldier  (Department  of  the  Army,  2005)  and  emphasizes 
the  requirement  for  Soldiers  to  create  and  adapt  strategies 
within  the  operational  environment  in  order  to 
successfully  execute  a  wide  range  of  tasks.  The  document 
emphasizes  that  for  current  and  future  deployments, 
psychological  and  cognitive  readiness  is  as  important  as 
physical  preparedness  (Department  of  the  Army,  2005,  p. 
140).  FOC  11-01,  Human  Engineering  for  the  Soldier, 
directs  the  Army  to  ensure  that  Soldiers  can  make 
decisions  reliably  and  effectively,  under  high  workload 
and  other  high  stress  conditions,  when  Future  Force 
performance  demands  will  be  the  greatest.  FOC  11-02, 
Man-Machine  Interface,  mandates  that  appropriate  tools 
and  procedures  be  used  to  understand  and  predict  the 


impact  of  Future  Force  Doctrine-Organization-Training  - 
Material-Leadership-Personnel-Facilities  (DOTMLPF) 
changes  on  Soldier  performance.  The  present  study 
supports  these  requirements,  as  its  focus  is  on  determining 
the  tools  that  can  maximize  the  adaptability  of  Officers 
and  enlisted  Soldiers  by  increasing  their  cognitive 
readiness  and  performance  within  a  complex  decision¬ 
making  environment,  in  preparation  for  the  real  events. 

In  support  of  the  Learning  with  Simulation  and 
Training  (LAST)  Army  Technology  Objective  (ATO),  the 
Army  Research  Laboratory  (ARL)  Human  Research  and 
Engineering  Directorate  (HRED)  investigated  how  select 
cognitive  and  affective  assessment  surveys  may  be 
predictive  of  decision-making  performance  in  a  training 
simulation.  Surveys  found  to  predict  performance  will  be 
recommended  for  inclusion  in  future  training  simulations 
(e.g.,  LAST  ATO)  as  a  way  to  inform  instructors,  or  in¬ 
game  virtual  coaches,  of  the  presumed  cognitive  readiness 
of  trainees  and  the  recommended  tailored  feedback. 
Cognitive  readiness  is  defined  as  the  optimization  and 
enhancement  of  human  cognitive  performance  (Foster, 
2001),  enabling  one  to  meet  the  expected  and  unexpected 
cognitive  demands  of  a  situation.  Consistent  with  the 
GEL  model  of  instruction  (Kirschner,  Sweller,  &  Clark, 
2006),  and  social  psychological  research  (Bandura  & 
Locke,  2003;  Cosenzo,  Fatkin,  &  Branscome,  2005; 
Fatkin  &  Hudgens,  1994),  providing  virtual  coaches  with 
information  about  a  user’s  current  cognitive  state  may 
structure  the  feedback  in  such  a  way  as  to  provide  the  user 
with  guidance  about  how  to  monitor  his  performance  as 
well  as  about  how  to  adjust  his  behavior  with  appropriate 
countermeasures. 

The  ARL  HRED  cognitive  readiness  assessment 
tools  and  standardized  procedures  provide  the  capability 
for  multidimensional  performance  assessment  and 
prediction  (Cosenzo,  Fatkin,  &  Patton,  in  press;  Fatkin, 
1998,  2003;  Fatkin,  Patton,  Mullins,  &  Burton,  2000). 
The  established  methodology  assesses  psychological 
resilience  and  cognitive  preparedness  levels  associated 
with  complex  decision  making  in  the  context  of  hostile 
and  rapidly  changing  environments  and  tasks.  The  ARL 
HRED  battery  of  cognitive  readiness  measures  has  been 
used  in  previous  research  investigations  to  identify 
characteristic  individual  strengths  and  vulnerabilities,  as 
well  as  to  quantify  cognitive  factors  that  mediate 
performance. 
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Research  has  demonstrated  that  individual 
differences  in  trait  characteristics,  such  as  personality, 
impact  performance  in  various  real  environments  (e.g., 
Cosenzo,  et  al.,  2005;  Fatkin,  2003;  Torre  et  al.,  1991; 
Mullins  &  Fatkin,  1999;  Schell,  Woodruff,  Corbin,  & 
Melton,  2005).  Torre  et  al.  (1991)  identified  personality 
factors  that  contributed  to  marksmanship  performance. 
Performing  cluster  analyses  on  participants’  Multiple 
Affect  Adjective  Check  List  -  Revised  (MAACL-R; 
Lubin  &  Zuckerman,  1999)  personality  characteristics 
during  a  marksmanship  competition  showed  two  distinct 
clusters  of  individuals.  One  group  showed  a  high  stability 
profile  (Low  Trait  Negative  Affect;  High  Trait  Positive 
Affect),  while  the  other  group  showed  a  relatively  low 
stability  profile  (High  Trait  Negative  Affect;  Low  Trait 
Positive  Affect).  Individuals  with  a  high  stability  profile 
performed  significantly  better  during  the  competition  than 
those  with  a  low  stability  profile. 

Similarly,  results  from  a  cluster  analysis  of  the 
MAACL-R  trait  subscales  obtained  from  Army  recruiters 
revealed  two  subgroups  with  the  same  distinct  profiles 
(Fatkin,  Mullins,  &  Patton,  1997).  Recruiters  in  the  high 
stability  group  were  significantly  more  successful  than 
were  those  in  the  low  stability  group,  accomplishing  91% 
compared  to  69%  of  their  recruitment  goals. 

In  addition  to  the  high  stability  traits  being  significant 
predictors  of  performance,  situational  self-efficacy  has 
also  proven  to  be  a  key  predictor  of  performance 
(Bandura  &  Locke,  2003;  Cosenzo  et  al.,  2005;  Fatkin  & 
Fludgens,  1994;  Fludgens,  Malkin,  &  Fatkin,  1992). 
Individuals  continually  assess  their  range  of  capabilities 
and  use  these  assessments  to  guide  and  influence 
subsequent  behavior  (Bandura,  &  Locke,  2003).  If 
individuals  perceive  their  capabilities  as  limited,  they  will 
tend  to  minimize  their  efforts,  perform  less  effectively,  or 
avoid  relatively  new  situations.  Those  who  reported  high 
self-efficacy  under  conditions  of  uncertainty  or  stress  are 
less  fearful  and  less  sensitive  to  criticism,  have  high 
energy  levels,  and  have  a  preference  for  hard  or 
challenging  work  (Cosenzo  et  al.,  in  press).  One  would 
expect  such  individuals  to  be  more  adaptive  in  changing 
and  unpredictable  situations. 

Individuals  differentially  employ  coping  mechanisms 
in  high  stress  situations.  The  Uncertainty  Response  Scale 
(URS)  specifically  assesses  individual  differences  in 
coping  with  uncertainty  (Bar-Tal  &  Spitzer,  1999;  Greco 
&  Roger,  2001).  Cosenzo  et  al.  (2005)  used  the  URS  to 
assess  coping  mechanisms  of  dispatchers  making  critical 
decisions  in  an  Emergency  Operations  Center.  Those 
individuals  with  a  high  Desire  for  Change  (a  high 
enjoyment  for  uncertainty,  novelty,  and  change)  and  high 
Cognitive  Uncertainty  (individual’s  need  to  plan  ahead, 
gather  information  and  seek  clarification)  completed  calls 


faster  than  those  with  low  scores  on  these  subscales  on  the 
URS. 

The  goal  of  the  present  study  was  to  understand  the 
predictive  relationships  between  one’s  cognitive 
assessment  responses  (i.e.,  affect,  confidence,  coping 
style,  and  personality  traits)  and  one’s  decision-making 
patterns  in  uncertain  situations  presented  within  the 
context  of  the  simulation,  Gator  6  -  Battery  Command 
Virtual  Experience  Immersive  Learning  Simulation™” 
(Gator  6;  WILL  Interactive,  2005).  Understanding  how 
decision-making  patterns  are  related  to  personality  traits 
and  coping  styles  will  allow  training  developers  to 
provide  customized,  interactive  mentoring  and  decision 
aids  to  users.  Although  empirical  studies  are  required  to 
confirm  our  expectations,  we  posit  that  a  customized 
training  experience  would  likely  increase  students’ 
confidence  and  personal  motivation  while  also  improving 
their  performance  on  key  learning  objectives  within  the 
simulation  (Morris,  Hancock,  &  Shirkey,  2004). 

Based  on  the  previous  research  described  above  and 
their  perceived  relevance  for  decision-making  in  uncertain 
situations,  we  chose  the  following  cognitive  assessment 
measures  from  a  set  of  standard  tools  used  by  ARL 
HRED:  (1)  Demographic  Questionnaire  (Fatkin  & 
Hudgens,  1994),  (2)  Life  Events  Form  I  (Fatkin  & 
Hudgens,  1994),  (3)  Zuckerman-Kuhlman  Personality 
Questionnaire,  Form  III  (Zuckerman  et  al.,  1993),  (4  &  5) 
MAACL-R  General/Trait  and  Today/State  forms  (Lubin 
&  Zuckerman,  1999),  (6)  Revised  Ways  of  Coping 
Checklist  (Vitaliano,  Maiuro,  Russo,  &  Becker,  1985),  (7) 
Need  for  Cognitive  Structure  (Bar-Tal,  1994),  (8)  Ability 
to  Achieve  Cognitive  Structure  (Bar-Tal,  1994),  (9)  URS 
(Greco  &  Roger,  2001),  (10)  Situational  Self-Efficacy 
scale  (Bandura,  1977),  (11)  National  Aeronautics  and 
Space  Administration  Task  Load  Index  (Sanders,  & 
McCormick,  1993),  and  (12)  Gator  6  Exit  Survey  (Dixon, 
Patton,  Fatkin,  Grynovicki,  and  Hernandez,  2005).  In  the 
present  paper,  we  will  present  only  the  results  related  to 
the  MAACL-R  Trait  and  State  forms,  the  URS,  the  SSE, 
and  the  Exit  Survey. 

Our  a  priori  hypotheses  were  as  follows: 

(1)  Participants  who  score  higher  on  the  Sensation- 
Seeking  scale  and  those  who  score  lower  on  the  Anxiety 
scale  of  the  MAACL-R  Trait  will  obtain  higher  scores  on 
mission-critical  decisions  in  Gator  6. 

(2)  Participants  who  report  lower  dysphoria  and 
higher  Positive  Affect  on  the  MAACL-R  Trait  will  obtain 
higher  scores  on  mission-critical  decisions  in  Gator  6. 

(3)  Participants  who  report  higher  confidence  in  their 
decisions  as  measured  by  the  SSE  scale  will  obtain  higher 
scores  in  Gator  6. 
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(4)  Participants  who  report  a  lower  emotional  (EU) 
and  cognitive  uncertainty  (CU)  and  a  higher  desire  for 
change  (DFC),  as  measured  by  the  URS,  will  obtain 
higher  scores  on  mission-critical  decisions  in  Gator  6. 

(5)  Participants  who  report  changes  in  positive  or 
negative  affect  at  specific  decision  points,  compared  to 
baseline  measures  (MAACL-R  State),  will  report  higher 
immersion  within  the  Gator  6  training  simulation  (Gator  6 
Exit  Survey). 

2.  METHOD 

This  section  provides  detailed  information  about  the  study 
participants,  the  Gator  6  simulation,  and  the  survey 
instillments. 

2.1.  Participants. 

Thirty-two  male  Captains  enrolled  in  the  U.S.  Army 
Field  Artillery  Captain’s  Career  Course  (CCC)  at  Fort 
Sill,  OK,  voluntarily  participated  in  this  study.  After 
investigators  briefed  the  CCC  on  the  purpose  and  tasks  of 
the  study,  all  willing  volunteers  signed  a  Volunteer 
Agreement  Affidavit  (VAA).  Investigators  and  the  VAA 
informed  participants  that  their  data  would  be  confidential 
and  that  they  had  the  right  to  withdraw  from  the  study  at 
any  time. 

2.2.  Gator  6  Simulation. 

As  part  of  its  leadership  seminar,  the  CCC  at  the 
Field  Artillery  School,  Fort  Sill,  OK,  routinely  uses  the 
computer-based  training,  Gator  6  (WILL  Interactive,  Inc., 
2005).  Gator  6  is  an  interactive  branching  video,  which 
prompts  trainees  to  make  decisions  while  role-playing  the 
lead  character,  CPT  Martin,  a  Battery  Commander  being 
deployed  to  the  Middle  East.  The  storyline  of  Gator  6  is 
based  on  actual  experiences  captured  in  interviews  with 
Soldiers  who  served  in  Operations  Iraqi  Freedom  and 
Enduring  Freedom.  Although  each  trainee  is  exposed  to 
the  same  questions  throughout  the  simulation,  their 
decisions  determine  the  version  of  the  next  video  that  is 
presented. 

Gator  6  is  provided  on  two  disks  and  is  divided  into 
seven  chapters.  Disk  1  consists  of  five  chapters:  1st  Day 
Pre-deployment,  2nd  Day  Pre-deployment,  14th  Day  Pre¬ 
deployment,  Arriving  in  Country,  and  Going  to  War. 
Disk  2  includes  two  chapters,  both  of  which  focus  on 
decision-making  in  post-combat  Stability,  Security, 
Transition  and  Reconstruction  (SSTR)  operations  where 
CPT  Martin’s  primary  mission  is  to  establish  a  democracy 
in  a  fictitious  Middle-Eastern  village.  The  purpose  of  the 
training  tool  is  to  prepare  junior  Officers  for  combat  and 
non-combat  decision-making  and  to  hone  their  leadership 


skills  through  virtual  practice.  We  chose  the  Gator  6 
simulation  because  it  is  used  in  an  Army  schoolhouse  and 
requires  students  to  make  decisions  in  both  combat  and 
post-combat  operations,  creating  an  environment  of 
uncertainty  similar  to  the  intended  focus  of  the  LAST 
ATO  prototype  simulation. 

2.3.  Survey  Instruments. 

As  described  earlier,  we  discuss  only  the  Trait  and 
State  MAACL-R,  the  SSE,  the  URS,  and  the  Gator  6  Exit 
Survey  in  the  present  paper. 

Multiple  Affect  Adjective  Check  List  -  Revised 
(MAACL-R),  General  &  Today  Form  (Lubin  & 
Zuckerman,  1999).  The  MAACL-R,  General  (Trait)  and 
Today  (State)  Forms  consist  of  five  primary  subscales: 
Anxiety  (response  to  emotional  uncertainty),  Depression 
(ceaseless  striving  or  a  sense  of  self  failure),  Hostility 
(frustration),  Positive  Affect  (positive  mood  or  well¬ 
being),  and  Sensation  Seeking  (sense  of  adventure), 
derived  from  a  one-page  check  list  of  132  adjectives.  An 
overall  distress  score,  Dysphoria  or  Negative  Affect,  is  a 
composite  of  the  Anxiety,  Depression,  and  Hostility 
scores.  The  only  difference  between  the  Trait  and  State 
forms  are  the  instructions  provided  to  participants.  Each 
form  requires  only  one  to  two  minutes  to  complete. 
Respondents  on  the  Trait  form  were  instructed  to  check 
all  the  words  that  describe  how  they  "generally"  feel, 
whereas  respondents  on  the  State  form  were  instructed  to 
check  all  of  the  words  that  describe  how  they  feel  “right 
now”  or  “since  the  last  time  you  completed  this  form.” 
Knowledge  of  the  specific  stress  components  assists  in  a 
more  appropriate  assignment  of  effective 
countermeasures.  We  administered  the  MAACL-R  Trait 
form  only  one  time  before  the  completion  of  the  Gator  6 
simulation,  while  we  administered  the  MAACL-R  State 
form  32  times  at  mission-critical  decision  points. 

The  Situational  Self-Efficacy  (SSE)  Scale  (B andura, 
1977,  1995,  1997).  The  SSE  Scale  investigates  the 
predictive  power  of  efficacy  expectations  about  behavior 
or  task  performance.  It  provides  an  assessment  of  one’s 
ability  to  master  new  situations  or  to  adapt  to  changing 
circumstances.  This  ability  is  considered  to  be  a 
composite  of  past  successful  and  failed  experiences  and  is 
associated  with  higher  levels  of  motivation  and 
performance  for  both  civilian  and  military  populations 
(Fatkin  &  Hudgens,  1994;  Potosky,  2002;  Sherer, 
Maddux,  Mercandante,  Prentice-Dunn,  Jacobs  &  Rogers, 
1982).  Participants  rated,  on  a  scale  of  1  -  10,  either  how 
confident  they  were  in  decisions  they  were  about  to  make 
(before  each  chapter)  or  in  decisions  that  they  just  made 
(during  and  after  each  chapter).  We  administered  the  SSE 
Scale  25  times  during  the  Gator  6  simulation  at  mission- 
critical  decision  points. 
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Uncertainty  Response  Scale  (URS;  Greco  &  Roger, 
2001).  The  URS  is  a  48-item  scale  designed  to  predict 
individual  differences  in  coping  with  uncertainty.  The 
Uncertainty  Response  Scale  is  comprised  of  three  factors: 
Emotional  Uncertainty  (EU),  Desire  for  Change  (DFC), 
and  Cognitive  Uncertainty  (CU).  Participants  rate 
statements  on  the  degree  to  which  each  statement  relates 
to  them  using  a  5-point  scale:  1  =  Never;  5  =  Always. 
Scores  for  subscales  are  determined  by  totaling  the  point 
value  of  statements  associated  with  each  subscale.  Higher 
scores  indicate  a  greater  tendency  toward  maladaptive 
responses  to  uncertainty  (EU),  greater  enjoyment  of  the 
unknown  (DFC),  and  greater  preference  for  control  under 
uncertain  conditions  (CU).  We  administered  the  URS 
only  one  time  before  the  completion  of  the  Gator  6 
simulation. 

Gator  6  Exit  Survey  (Dixon  et  al.,  2005),  This  survey 
contains  18  questions  about  the  structure,  content,  and 
quality  of  the  Gator  6  training  simulation,  as  well  as 
questions  about  participants’  perceived  engagement  in  the 
simulation  scenarios.  Most  of  the  questions  required 
forced-choice  responses,  with  5  choice-points.  The 
anchors  varied  by  question  (e.g.,  Extremely 
Unclear/Difficult,  Extremely  Clear/Easy). 

2.4.  Procedure. 

Prior  to  participation  in  the  study,  investigators 
briefed  the  CCC  on  the  purpose  and  tasks  of  the  study  and 
read  the  Volunteer  Agreement  Affidavit  (VAA)  to  them. 
Because  completion  of  the  Gator  6  simulation  was  a 
requirement  of  the  CCC,  the  investigators  explained  to 
potential  study  volunteers  that  their  completion  of  the 
surveys  before,  during,  and  after  the  completion  of  the 
Gator  6  simulation  was  strictly  voluntary  and  that  they 
could  withdraw  from  the  experiment  at  anytime. 
Investigators  further  explained  that  all  data  would  be 
stored  confidentially  with  a  participant  number,  used 
solely  for  research  purposes,  and  would  not  be  distributed 
to  anyone  in  their  chain  of  command. 

At  the  end  of  the  briefing,  investigators  distributed 
the  VAA  and  Trait  measures  in  separate  folders  to  each  of 
the  potential  volunteers  and  asked  that  all  who  agreed  to 
participate  to  sign  the  VAA  and  to  remember  the  number 
written  on  their  folders  (i.e.,  their  participant  number). 
Those  who  chose  not  to  participate  returned  a  folder  with 
the  uncompleted  forms  to  the  investigators,  thereby  not 
being  singled  out  amongst  the  group.  After  all  volunteers 
completed  the  Trait  surveys  (approximately  one  hour), 
investigators  took  participants  to  the  designated  computer 
lab  at  Fort  Sill  and  explained  the  Gator  6  simulation  and 
the  associated  surveys.  Each  participant  sat  at  his  own 
computer  workstation  and  wore  headphones  to  hear  the 
audio  of  the  narrator  and  video  clips.  At  each  participant 
workstation,  investigators  placed  eight  separate  folders 


containing  the  surveys  (one  for  each  of  the  seven  chapters 
and  one  for  the  final  surveys).  The  total  time  participants 
required  for  the  surveys  and  the  Gator  6  simulation  was 
approximately  seven  hours. 

Participants  completed  the  MAACL-R  Trait  and  the 
URS  once  before  completing  Gator  6.  They  completed 
the  State  MAACL-R  before,  during,  and  after  each 
chapter  as  highlighted  in  section  2.2.  Participants 
completed  the  State  MAACL-R  surveys  at  strategic 
decision-making  points,  indicated  by  messages  ( Please 
complete  the  surveys.)  within  the  simulation  after  specific 
mission-critical  decision  points.  The  dependent  variable 
for  Gator  6  performance  was  the  percentage  of  correct 
responses  across  mission-critical  decisions.  Participants 
completed  the  Gator  6  Exit  Survey  after  they  finished  the 
Gator  6  simulation.  Participants  completed  the  Gator  6 
training  simulation  in  a  self-paced  manner,  without  their 
instructors  present,  and  investigators  permitted 
participants  to  leave  as  they  finished.  Investigators  were 
present  during  the  simulation  to  monitor  the  completion 
of  surveys  and  to  answer  participants’  questions. 

3.  RESULTS 

The  results  of  the  study  conducted  at  the  Field 
Artillery  Captain’s  Career  Course  are  consistent  with 
previous  research  conducted  by  ARL  HRED  in  real 
operational  environments.  The  findings  suggest  that  the 
responses  on  the  trait  and  state  cognitive  readiness 
measures  such  as  trait  and  state  affect,  situational  self- 
efficacy,  and  coping  with  uncertainty,  capture  information 
about  individual  factors  that  may  influence  performance. 

Although  the  Trait  MAACL-R  scores  were  not 
predictive  of  overall  performance  on  the  Gator  6 
simulation,  changes  in  affect  (State  MAACL-R)  during 
the  simulation  were  correlated  with  specific  events  in  the 
simulation,  denoting  immersion.  Positive  correlations 
were  also  found  between  the  Trait  MAACL-R  scores  and 
the  Uncertainty  Response  Scale  (URS),  and  the  Desire  for 
Change  (DFC)  measure  of  the  URS  predicted  overall 
Gator  6  performance. 

3.1.  Trait  MAACL-R  Scores  (General). 

Table  1  shows  the  correlations  between  participants’ 
Trait  MAACL-R  scores  and  their  scores  on  the  URS 
subscales  of  Emotional  Uncertainty  (EU)  and  DFC. 
Responses  on  the  EU  subscale  were  positively  correlated 
with  trait  depression  (TDEP),  hostility  (THOS),  and 
dysphoria  (TDYS),  and  they  were  negatively  correlated 
with  Trait  positive  affect  (TPA).  As  one  might  expect, 
this  indicates  that  individuals  with  more  overall  negative 
affect  also  have  higher  levels  of  emotional  uncertainty, 
while  the  opposite  is  true  for  those  with  a  generally 
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optimistic  outlook.  Responses  on  the  DFC  were 
negatively  correlated  with  Trait  depression  and  dysphoria, 
indicating  individuals  with  higher  overall  negative  affect 
also  have  less  of  a  desire  for  change. 


Table  1.  Correlations  between  trait  MAACL-R  scores 
and  scores  on  the  URS  subscales,  EU  &  DFC. 


Measure 

TDEP 

THOS 

TPA 

TDYS 

EU 

0.499* 

0.690* 

-0.542* 

0.689** 

DFC 

-0.408* 

— 

— 

-0.411* 

*p<  0.05;  **  p<  0.01 


3.2.  State  MAACL-R  Scores  and  Immersion  within  the 
Simulation. 

To  evaluate  stress  or  affect  levels  of  the  participants 
during  the  simulation  (state  collections),  we  administered 
the  State  MAACL-R.  Because  there  were  two  completely 
separate  modules,  one  on  each  disk,  we  evaluated  the 
stress  levels  separately  according  to  disk  and  chapter.  For 
each  of  the  two  disks,  we  conducted  a  three-way 
Multivariate  Analysis  of  Variance  (MANOVA),  Chapter 
x  Sessions  x  MAACL-R  Subscales,  to  look  at  the  levels 
of  affective  response  throughout  the  simulation.  Changes 
in  the  affective  responses  of  participants  are  suggestive  of 
immersion  within  the  simulation  scenario,  as  participants 
identify  themselves  as  CPT  Martin  and  the  decisions  with 
which  he  is  faced  over  time. 

The  MANOVAs  for  each  disk  and  chapter  revealed 
significant  differences  within  chapters  as  a  function  of  the 
State  MAACL-R  responses.  Only  Disk  1,  Chapter  1 
showed  significant  results  for  the  Disk  x  Chapter  x 
Measure  interaction  (Wilks  X  =  .379,  F(2,54)  =  3.278,  p  < 
0.05).  Table  2  lists  the  significant  main  effects  for  the 
MAACL-R  measured  by  disk  and  chapter. 

Table  2.  MANOVA  results  for  the  state  MAACL-R 
x  chapter  interaction. 


Disk  1 

-  Chapter  1  (Pre-Deployment  Day  1) 

Wilks  X  =  0.745,  F(2,54)  =  4.786 ,p  <  0.05 

Disk  2 

-  Chapter  1  (SSTR  Operations) 

Wilks  X  =  0.550,  F(2,54)  =  2.722,  p  <  0.05 


Significant  differences  in  positive  affect  for  Disk  1 
were  as  follows:  PreDICl  vs.  PostDICI  (t  =  3.008,  p  < 
0.01),  as  well  as  for  PreDICl  vs.  DurlDICl  (t  =  2.736,  p 
<  0.01);  PreDlC4  vs.  PostDlC4  (t  =  4.807,  p  <  0.001), 
PreDlC4  vs.  both  DurlDlC4  and  Dur2DlC4  (t  =  3.585 ,p 
=  0.001;  t  =  4.980,  p  <  0.001).  These  results  highlight 


changes  in  positive  affect  in  response  to  particular  events 
within  Chapters  1  (difficult  pre-deployment  decisions) 
and  4  (decisions  under  time  pressure)  of  Disk  1 . 

Significant  differences  in  Positive  Affect  found  on 
Disk  2  are  as  follows:  pre-D2Cl  vs.  Dur2,  3,  and  5D2C1 
(t  =  3.464,  p  <  0.01),  (t  =  3.425,  p  <  0.01);  (t  =  2.801  ,p< 
0.01).  These  differences  refer  to  changes  in  positive 
affect  in  response  to  particular  events  that  occurred  within 
Chapter  1  (unfamiliar  SSTR  operations)  of  Disk  2,  which 
further  support  the  correlations  denoting  immersion  in  the 
simulation  (see  Table  3). 

Table  3  shows  that  participants’  positive  affect  scores 
on  the  State  MAACL-R  were  correlated  with  their 
responses  to  a  question  on  the  Exit  Survey  about  their 
ability  to  assume  the  role  of  the  main  character  within  the 
simulation,  CPT  Martin. 

Low  levels  of  positive  affect  indicate  the  individual  is 
unsure  of  his  ability  or  resources  for  coping  with  difficult 
or  threatening  circumstances.  As  the  participants  made 
pre-deployment  command  decisions  (Disk  1)  as  well  as 
decisions  within  unfamiliar  SSTR  operations  (Disk  2), 
while  assuming  the  role  of  CPT  Martin,  they  experienced 
a  decrease  in  positive  affect  levels  or  sense  of  well-being. 
The  levels  were  similar  to  officers  who  participated  in 
toxic  agent  chemical  decontamination  training  at  Fort 
McClellan,  AL  (Fatkin  &  Fludgens,  1994). 


Table  3.  Immersion:  Negative  correlations  between  the 
MAACL-R  positive  affect  subscale  and  participants’ 
ability  to  assume  the  role  as  CPT  Martin. 


Measure 

CPT  Martin 

PreDICIPA 

-0.392* 

DurDICIPA 

-0.388* 

Dur2DlC3PA 

-0.481** 

PostDlC3PA 

-0.461* 

DurlDlC4PA 

-0.452* 

*p<  0.05;  **  p<  0.01 


Another  indication  of  immersion  within  the 
simulation  was  significant  changes  (p  <  0.05)  in 
participants’  State  MAACL-R  hostility  levels  during  Disk 
2,  Chapter  2.  Also,  CCC  participants  showed  hostility 
levels  comparable  to  participants  with  moderate  to  high 
hostility  in  previous  studies  (Torre  et  al.,  1991;  e.g. 
marksmanship  competition  &  students  taking  a  medical 
exam).  The  hostility  levels  here  may  be  indicative  of  the 
mismatch  between  the  available  and  expected  decisions 
choices  during  the  simulation  or  due  to  their  frustration 
with  the  SSTR  operations. 
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3.3.  Situational  Self-Efficacy  (SSE). 

Cluster  analyses  group  individuals  that  show  similar 
characteristics  and  minimize  the  variance  for  each  cluster 
across  the  measures.  Cluster  analyses  conducted  on  the 
situational  self-efficacy  scores  across  participants 
revealed  two  subgroups  of  individuals  with  distinct  levels 
of  self-efficacy,  a  “high  SSE”  group  and  a  “moderate 
SSE”  group.  With  a  potential  range  of  self-efficacy 
scores  from  1  to  10,  the  “high  SSE”  group  rated 
themselves  as  having  an  efficacy  level  of  10.  The 
“moderate  SSE”  group  reported  a  mean  level  of  efficacy 
of  8.5.  The  F-statistic  for  cluster  was  significant,  ,F(1,28) 
=  63.716,  p  <  0.001,  indicating  that  self-efficacy  provided 
a  critical  contribution  to  the  evaluation  of  individual 
variability. 

3.4.  Desire  for  Change  (DFC)  or  Adaptability. 

We  also  predicted  that  individuals  comfortable  with 
uncertainty  and  change,  as  measured  by  the  URS,  should 
perform  better  on  mission-critical  decisions  during  the 
simulation.  As  shown  in  Figure  1,  Pearson  correlations 
and  a  linear  regression  analysis  revealed  that  the  DFC 
subscale  was  the  only  significant  predictor  of  overall 
Gator  6  performance,  as  measured  by  percentage  of 
correct  decisions  (R2=0.19;  F(l,27)=  6.29,  p  <  0.05): 

(1)  Overall  Gator  6  Score  =  37.00  +  0.62  *  DFC  score. 
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Figure  1.  Overall  Gator  6  performance  increases  as  a 
function  of  trait  desire  for  change  score. 


As  predicted,  individuals  with  higher  scores  on  the 
DFC  subscale  showed  better  decision-making 
performance,  implying  that  individuals  who  enjoy  change, 
or  who  are  adaptable,  seem  better  able  to  make  decisions 


in  the  context  of  uncertainty.  Moreover,  correlations 
between  the  MAACL-R  and  the  DFC  showed  that 
individuals  with  lower  trait  negative  affect  and  lower  trait 
perceived  failure  had  a  higher  desire  for  change  (refer  to 
Table  1). 


4.  DISCUSSION 

The  results  of  this  study  show  that  some  cognitive 
readiness  measures  can  help  to  explain  participants’ 
performance  within  a  dynamic  simulation  in  which  users 
make  decisions  under  great  uncertainty.  We  propose  the 
integration  of  a  Cognitive  Assessment  Support  Tool 
(CAST)  to  drive  feedback  within  simulations. 

Training  developers  can  use  the  information  from  the 
cognitive  readiness  metrics  to  design  customized, 
interactive  training  aids  for  assisting  users  in  meeting 
training  objectives.  Chung,  Baker,  Delacruz,  et  al.  (2006) 
assert  there  is  a  current  revolution  in  training  and  call  for 
research-based  human  performance  and  assessment 
knowledge  and  tools.  One  mechanism  proposed  by  ARL 
FIRED  (Dixon,  2006)  for  incorporating  the  proactive 
assistance  of  this  tailored  feedback  is  the  Cognitive 
Assessment  Support  Tool  (CAST). 

CAST  would  be  employed  for  guiding  users  through 
critical  scenario  progress  points  using  real-time  feedback. 
It  is  based  on  theoretically  sound  constructs  regarding 
personality  traits  and  confidence  levels  as  predictors  of 
performance  under  conditions  of  uncertainty.  These 
constructs  are  woven  throughout  the  creative  development 
and  subsequent  implementation  of  the  tailored  feedback 
mechanism. 

For  example,  ratings  of  self-confidence  in  the  ability 
to  perform  well  and  make  sound  decisions  are  measured 
by  the  Situational  Self-Efficacy  scale  (SSE).  The  SSE 
ratings  obtained  at  critical  points  throughout  the 
simulation  would  provide  information  on  scenario-to- 
scenario  shifts  in  confidence  levels.  These  shifts  or  deltas 
would  serve  as  input  for  initiating  the  CAST  feedback. 

Descriptions  of  personality  profiles  obtained  from  the 
cognitive  readiness  metrics  before  the  start  of  the 
simulation  could  provide  the  basis  for  the  timing, 
integration,  and  content  of  within-game  feedback. 

Brief,  strategically  placed  “quizzes”  written  into  the 
game  scenario  could  provide  a  source  of  information  on 
the  user’s  confidence  state.  The  CAST  would  be  initiated 
after  computing  a  significant  change  in  the  selected 
cognitive  readiness  metrics  over  a  short  period  of  time. 
For  example,  we  know  that  high  SSE  ratings  (8.5  to  10) 
are  significantly  and  positively  associated  with  cultural 
awareness  and  effective  leadership  behavior.  Therefore,  a 
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significant  drop  in  SSE  ratings  just  prior  to  a  critical 
related  event  would  trigger  the  initiation  of  feedback 
tailored  to  the  interaction  needed  between  the  individual 
and  the  situation. 

The  CAST  would  trigger  the  tutoring  system  to 
provide  constructive  feedback  to  the  user  as  determined 
by  the  interaction  between  the  personality  profile  of  the 
user,  the  scenario  parameters,  and  the  expected  outcomes. 
For  example,  a  user  with  a  trait  profile  reflecting  low 
“desire  for  change”  would  be  uneasy  with  uncertainty, 
and  may  become  anxious  in  the  midst  of  novel  situations. 
When  approaching  a  new,  high  stakes  situation,  CAST 
could  detect  an  overcautious  approach  (based  on  extended 
decision-making  time)  and  would  be  prompted  to  provide 
the  user  with  helpful  information  (e.g.,  a  suggested 
prioritization  of  information  the  user  already  has;  a  coping 
strategy  to  allow  the  user  to  calm  down  and  make  a 
rational  decision;  etc.).  CAST  would  have  the  capability 
to  present  feedback  to  the  user  only  when  needed,  as 
opposed  to  providing  feedback  at  every  critical  decision 
point. 

The  CAST  could  detect  a  mismatch  or  incongruence 
between  user  characteristics,  user  abilities,  and  the 
requirements  for  successful  performance.  Customized, 
interactive  mentoring  provided  at  critical  points  within  the 
simulation  would  serve  to  enhance  the  individual's  ability 
to  accomplish  the  learning  objectives. 

We  posit  that  the  CAST  adds  a  unique  component  to 
the  GEL  model  of  instruction.  Only  when  such  a 
framework  is  incorporated  into  the  tutoring  systems  of 
future  simulations,  such  as  future  products  developed 
within  the  LAST  ATO,  could  we  test  the  hypothesis  that 
this  tailored  approach  provides  a  more  effective  means  for 
the  transfer  of  skills  learned  during  training  simulations. 
By  incorporating  personal  characteristics  and  cognitive 
strategies  customized  to  the  individual’s  cognitive  or 
affective  needs  or  coping  style,  skills  will  more  likely  be 
generalized  to  novel  settings,  particularly  under 
conditions  of  uncertainty  and  unpredictability  that  require 
adaptive  decision-making  (Bandura  &  Locke,  2003; 
Driskell,  Johnston,  &  Salas,  2001). 

5.  CONCLUSIONS 

The  results  of  the  present  study  suggest  that  responses  on 
cognitive  and  affective  assessment  surveys  may  provide 
an  additional  indication  of  the  cognitive  readiness  of 
individual  users  as  they  participate  in  a  training 
simulation.  As  proposed  in  the  CAST  methodology,  such 
information  could,  in  turn,  be  utilized  by  intelligent 
tutoring  systems  to  structure  the  feedback  to  complement 
or  counteract  users’  current  state,  perceived  expectations, 
and  coping  styles.  Further  research  is  necessary  in  order 


to  incorporate  this  methodology  into  the  LAST  ATO 
training  software,  which  is  being  developed  in 
collaboration  with  the  Institute  for  Creative  Technologies, 
the  Simulation  Training  &  Technology  Center,  the  Army 
Research  Institute,  and  the  School  for  Command 
Preparation  at  Fort  Leavenworth  (Hill  et  al.,  2006). 
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